Add CI that builds and runs every example by aidangarske · Pull Request #598 · wolfSSL/wolfssl-examples

aidangarske · 2026-07-15T00:38:18Z

Adds CI that builds and runs every example against wolfSSL master and the latest stable tag, asserting on real output — not just exit 0 — and fixes the example bugs that blocked those runs from going green.

Issue-creation triage tested here: CI: hash failing in the nightly aidangarske/wolfssl-examples#2
Manifest coverage: 72 run, 37 build-only, 34 skipped (each with a reason) — 60 of the runs assert on output, not just exit 0.
History is one atomic commit per fix, so this can be reviewed fix-by-fix.

CI

Manifest-driven harness (run_example.py + examples-manifest.yml): build → run → assert expected output.
Per-target workflows for environments that need special toolchains: ESP32, Android, RT1060, UEFI, SGX, PSA/mbedTLS, device sims (SE050, ATECC608, STSAFE, TROPIC01, STM32, PIC32MZ), DTLS multicast, Windows/MSVC, cross-arch (m32/aarch64/s390x), sanitizers/valgrind/cppcheck, config-matrix.
Nightly runs everything vs master + stable and triages failures into per-directory GitHub issues (retry-once, then flake-vs-real classification).
Every network fetch (git clone, download, docker pull) is wrapped in a retry so a transient blip does not fail a job.

Fixes

Each is its own commit:

Expired fixtures — regenerated CRL + certs that silently broke the CRL/verify examples.
Wrong exit codes — servers/examples that "passed" by returning a byte count or an unconditional -1; they now fail when they should.
Never-worked examples — PQ/ML-DSA/ML-KEM API ports, a client-cert-less resume, identical XTS key halves.
Toolchain drift — ESP-IDF 5.x ENC28J60 + PHY ports, RT1060 cross-build, UEFI freestanding, Android NDK/API-31, mynewt 64-bit.
Real bugs — a custom-IO double-free/use-after-free, DTLS resume-on-close_notify + record-spanning, TLS 1.3 early-data session-ticket handling.

Upstream wolfSSL bugs this CI surfaced

Examples are marked known-fail and gated per ref with fixed_on; drop the marker when each merges:

OCSP stapling — a CTX-set status_request is dropped from the TLS 1.3 Certificate message (-429): Check ctx extensions before rejecting a TLS1.3 certificate message extension wolfssl#10936 (merged to master)
PKCS#11 — wc_ecc_verify_hash on a label-generated EC key returns WC_HW_E (-248): Set label and ID on PKCS11 EC public key wolfssl#10954 (in review)
PKCS#7 — smime/indef flags: Fix PKCS7 streamed SignedData dropping content wolfssl#10904
point to ocsp and pkcs back to master after wolfssl merges

Copilot

Pull request overview

Adds a comprehensive GitHub Actions CI system to build and (where possible) run every example in this repository against wolfSSL master and the latest stable tag(s), using a manifest-driven matrix so failures are isolated to per-example jobs.

Changes:

Introduces manifest-driven CI workflows (host/emulated/cross) plus a nightly triage workflow that retries, classifies flakes vs real failures, and can file/update issues.
Adds Python harness/scripts to generate matrices, enforce “coverage” (no buildable example directory left unaccounted for), run examples with output assertions, and collect artifacts for triage.
Updates many examples/Makefiles/scripts/fixtures so they build/run reliably under CI (exit statuses, paths to certs, linking flags, regenerated CRL/certs, etc.).

Reviewed changes

Copilot reviewed 62 out of 64 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
X9.146/Makefile	Makefile header tweak for X9.146 examples
x509_acert/README.md	Fix build/run instructions and cert paths
x509_acert/Makefile	Make OpenSSL prefix optional; adjust lib paths
uefi-static/Makefile	Make libgcc/gnu-efi paths less distro-specific
tls/server-tls13.c	Ensure successful exit status after shutdown
tls/server-tls13-certauth-clienthello.c	Ensure successful exit status after shutdown
tls/server-tls-posthsauth.c	Ensure successful exit status after shutdown
tls/client-tls13-resume.c	Load client cert/key and CA separately
tls/client-tls.c	Load client cert/key and CA separately
staticmemory/memory-bucket-optimizer/tester/Makefile	Add include/lib paths for wolfSSL
staticmemory/memory-bucket-optimizer/optimizer/Makefile	Add include/lib paths for wolfSSL
staticmemory/Makefile	Correct Makefile title/comment
signature/sigtest/Makefile	Make OpenSSL path optional and clearer
SGX_Linux/sgx_t.mk	Update SGX C++ runtime library name
scripts/run-all-examples.sh	Add local manifest-driven runner script
psa/Makefile	Fix include path and guard empty PSA_INCLUDE
pq/stateful_hash_sig/xmss_example.c	Remove redundant include
pq/stateful_hash_sig/lms_example.c	Remove redundant include
pq/ml_kem/README.md	Switch build instructions to `make`
pq/ml_kem/ml_kem.c	Update include/header usage; simplify main signature
pq/ml_kem/Makefile	Add Makefile for ML-KEM example
pq/ml_dsa/ml_dsa.c	Update to FIPS 204 ctx APIs; fix exit code
pq/ml_dsa/Makefile	Fix clean target artifacts list
pkcs7/scripts/runall.sh	Fix ordering + ensure failures exit non-zero
pkcs7/scripts/openssl-verify.sh	Gate on runall, add failure accounting, improve checks
pkcs7/Makefile	Correct Makefile title and zlib comment
pkcs11/Makefile	Correct Makefile title/comment
pk/srp/Makefile	Fix WOLFSSL_INSTALL_DIR prefix
pk/enc-through-sign-rsa/rsa-public-decrypt-app.c	Normalize return code from length-returning API
embedded/Makefile	Adjust linux-specific target list handling
ecc/ecc-params.c	Handle negative lengths; normalize exit status
ebpf/tls-uprobe-trace/Makefile	Add wolfSSL include/lib flags
dtls/server-dtls-rw-threads.c	Fix cert paths for CI execution
dtls/client-dtls-rw-threads.c	Fix cert paths for CI execution
custom-io-callbacks/file-server/file-server.c	Fix server return codes / cleanup flow
custom-io-callbacks/file-server/check.sh	Make server/client orchestration bounded and reliable
custom-io-callbacks/file-client/file-client.c	Fix client return codes
crypto/keywrap/Makefile	Add Makefile for keywrap example
crypto/aes-modes/aes-xts.c	Fix XTS key halves to be distinct
certvfy/Makefile	Add include/lib flags for wolfSSL
certs/crl/regenerate-crl.sh	Add script to regenerate CRL fixture
certs/crl/crl.pem	Update CRL fixture to unexpired version
certmanager/README.md	Fix example invocation name
certmanager/certverify.c	Ensure successful exit status
certmanager/certloadverifybuffer.c	Update embedded cert fixtures
certgen/Makefile	Add Makefile header/comment
certgen/csr_w_ed25519_example.c	Normalize return code from length-returning API
can-bus/Makefile	Add include/lib flags for wolfSSL
can-bus/generate_ssl.sh	Cap CN length to X.509 limits
btle/common/btle-sim.c	Ignore SIGPIPE to avoid FIFO-close crashes
.gitignore	Ignore new binaries/pycache; fix typo
.github/workflows/nightly.yml	Add scheduled nightly orchestration
.github/workflows/examples.yml	Add host-tier matrix workflow + coverage gate
.github/workflows/emulated.yml	Add emulated-tier workflow (TPM/SGX/CAN/UEFI)
.github/workflows/cross-build.yml	Add cross-build workflow (ESP32/Pico/etc.)
.github/workflows/ci-triage.yml	Add triage workflow for nightly results
.github/workflows/_resolve-wolfssl.yml	Add reusable wolfSSL ref resolution workflow
.github/scripts/run_example.py	Add example harness (build/run/assert/output/artifacts)
.github/scripts/manifest.py	Add manifest loader + matrix generator + coverage checker
.github/scripts/issue_store.py	Add issue identity/body/state machine + scrubbing
.github/scripts/ci_triage.py	Add triage logic (retry, classify, issues, AI advisory)
.github/examples-manifest.yml	Add source-of-truth manifest for CI coverage/matrix
.github/actions/setup-wolfssl/action.yml	Add composite action to build/cache/install wolfSSL

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

padelsbach · 2026-07-22T16:16:18Z

General comment: I'd argue the build+test scripts should not live in the CI dir, and should not be shell scripts. make check for each applicable example would be more canonical and perhaps closer to the end-user environment. (And it's how Provider runs CI as well.)

Ofc, keep the ones that don't fit that model (eg Android)

…n BUFFER_E

… workflow_call

…cks and inputs

…ith ecc

…nputs into the check targets

…heck is not a false pass

…ep that BSD make lacks

… verify result

… pipefail grep so they hold under BSD make

Copilot AI review requested due to automatic review settings July 15, 2026 00:38

Copilot started reviewing on behalf of aidangarske July 15, 2026 00:38 View session

Copilot AI reviewed Jul 15, 2026

View reviewed changes

aidangarske force-pushed the ci-matrix branch 5 times, most recently from 56d67ec to f0f3a18 Compare July 15, 2026 16:34

aidangarske mentioned this pull request Jul 15, 2026

Retry the ISO TP flow control wait up to N_Bs wolfSSL/wolfssl#10915

Closed

aidangarske force-pushed the ci-matrix branch from 378a78a to ddcb1c4 Compare July 15, 2026 20:15

aidangarske self-assigned this Jul 16, 2026

aidangarske force-pushed the ci-matrix branch 19 times, most recently from 0ffccd5 to f4fd5e6 Compare July 16, 2026 22:54

padelsbach reviewed Jul 22, 2026

View reviewed changes

aidangarske added 10 commits July 22, 2026 09:48

Add a make check target to each applicable example

69db57a

Run only the example and lint smoke set on draft PRs

df4568c

Only run a per-target workflow when its own example dir changes

8c71295

Assert the real se050 wolfcrypt result instead of an early sub-test line

86f07c7

Size the RSA 2048 key export buffers so the UEFI test does not fail o…

a32df9f

…n BUFFER_E

Call the always-present MLDSA context API from the UEFI driver

9eb6617

Cross uefi-static and uefi-library with both wolfSSL refs in the matrix

3f3b0dd

Give each tpm matrix leg a ref-unique results file and artifact

a840e1f

Add a codespell spellcheck pass to the lint job

320e6c9

Run push CI on master only so a PR branch does not double-trigger

0601ffd

padelsbach reviewed Jul 22, 2026

View reviewed changes

aidangarske added 8 commits July 22, 2026 12:19

Select valgrind by caller_run_id since event_name is the caller under…

5ea19d3

… workflow_call

Make example check targets catch real failures with pipefail exit che…

5edb6f1

…cks and inputs

Wire the harness to run make check for mode check examples starting w…

2024add

…ith ecc

Migrate the single-entry exec examples to mode check and fold their i…

4871c14

…nputs into the check targets

Fix four make check assertions that misfired under pipefail

818d1cd

Give the tpm manifest entry a run step so it asserts output

c11774d

Skip uefi-static in the lint make -n loop so it does not clone

a629a78

Assert the actual verify result in the pkcs7 and rsa-nb checks

8b20297

padelsbach reviewed Jul 22, 2026

View reviewed changes

aidangarske added 7 commits July 22, 2026 17:40

Return nonzero from pkcs12-create-example on a failed create so the c…

3f97419

…heck is not a false pass

Return nonzero from rsa-kg on any key generate or write failure

78e628a

Assert the static memory checks by exit code instead of a pipefail gr…

fd9727c

…ep that BSD make lacks

Run pkcs7 signedData stream through make check so it asserts the real…

b302438

… verify result

Check DER certificates and CRLs in the expiry canary too

0b24d8a

Feed the wolfHSM client its stdin so run_client actually exchanges data

4086620

Assert every make check by exit code and captured output instead of a…

f86683f

… pipefail grep so they hold under BSD make

padelsbach approved these changes Jul 23, 2026

View reviewed changes

Uh oh!

Conversation

aidangarske commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI

Fixes

Upstream wolfSSL bugs this CI surfaced

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

padelsbach commented Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aidangarske commented Jul 15, 2026 •

edited

Loading